Automatic exposure time selection for imaging tissue

ABSTRACT

The invention relates to a system for automatically adjusting an exposure time to improve or otherwise optimize a dynamic range of a digital image. The system includes a camera configured to capture an image of a subject within the field of view at a first exposure time. The captured image is composed of multiple pixels, with each pixel having a respective intensity value. The system further includes a shutter or suitable control configured to control an exposure time of the camera. A controller configured to carryout the following steps including: (a) querying a frequency distribution of pixel intensity values; (b) determining an effective “center of mass” of such a distribution, or histogram, to determine an adjusted exposure time; and (c) capturing a second image of the subject at the adjusted exposure time thereby obtaining an image with an improved or optimal dynamic range.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/201,753, filed Aug. 29, 2008, which claims priority from U.S.Provisional Application No. 60/969,540, filed Aug. 31, 2007,incorporated herein by reference in their entirety.

FIELD

The present invention relates generally to the field of imaging ofbiological specimens i.e., histology specimens such as tissue microarray (TMA) “histospots” and whole tissue section (WTS) fields of view(FOV) for image analysis.

BACKGROUND

Platforms for the analysis of tissue-based quantitative proteinbiomarker assay studies and clinical diagnostics tend to have variableresults due to operator interaction. In such systems, an operation mayexamine histology specimens, otherwise referred to herein as samples,using a microscope system configured to capture a magnified image of thespecimens. Operator decision making and setup interactions, such asthose related to image capture, may lead to quantitative error in lateranalysis. For certain immunohistochemistry (IHC) image analysis, themagnified image is captured with a digital camera for which an exposuretime for image acquisition may be set manually.

Unfortunately, such a manual methodology introduces several significantlimitations. For example, the operator may select what appears to be thehighest expressing fields of view of a whole tissue section, or thehighest expressing histospots of a TMA to determine exposure times thatare then applied to all other images acquired during a particular test.Such an approach generally limits the overall dynamic range of an assayin that optimization of image acquisition for an expression level of asingle sample may be to the detriment of the other samples.Additionally, different users may determine different initial exposuretimes for application across samples in a particular study or differentfields of view or different histospots. In either instance, suchoperator-dependant variability introduces possible variations even whenusing the same system. Still further, there is a first operatorinteraction time related to examination of the sample and determinationof a field of view (histospots) and a second operator interaction timerelated to exposure time selection and performance of the actual setup.In order to realize the greatest benefit of automatically producedconsistent, quantitative data on an automated IHC analysis microscopyplatform, any interaction of the user with the system, including anumber of operator decisions, should be kept to a minimum.

SUMMARY

The systems and processes described herein provide automatic selectionof the appropriate exposure times for image acquisition using amicroscope system. Such automated selection can be repeated for allchannels or fluorescent signals being monitored in a fluorescencemicroscope based System. Alternatively or in addition, such automatedselection can be repeated for all fields being examined. In someembodiments, every field of view (or histospot) examined, the cameraexposure is optimized before each image is acquired with these exposuretimes re-optimized for each subsequent image. This procedure results ina high degree of correlation between data acquired using the processesof the invention with data acquired by a highly trained operator inmanual mode.

In one aspect, the invention relates to a process for automaticallyoptimizing a dynamic range of pixel intensity data obtained from adigital image. The process includes capturing an image of a subjectwithin the field of view of a camera at a first exposure time, resultingin a captured image comprising a predetermined number of pixels, whereineach pixel has an intensity value. A frequency distribution of pixelintensities of the captured image is queried to determine a region ofthe greatest frequency occurrence of the pixel intensities of thefrequency distribution. Exposure time is then adjusted from the firstexposure time to shift that region of highest frequency distributiontoward the middle of the range of intensity values. In other words, thecenter of mass of a histogram, or frequency distribution is determinedfrom which an adjusted exposure time is calculated to achieve anoptimized dynamic range of pixel intensities. A second image of thesubject can then be captured at the adjusted exposure time resulting inan image having an optimal dynamic range.

Accordingly, a process for automatically improving a dynamic range of adigital image is provided, which process includes: (a) capturing a firstimage of a subject within the field of view of a camera at a firstexposure time, resulting in a captured first image comprising apredetermined number of pixels, each pixel having an intensity value,(b) evaluating a frequency distribution of the pixel intensity values ofthe first image; (c) determining a region of highest frequencydistribution of the histogram to determine an exposure time adjustmentto shift the region of highest frequency distribution toward a center ofa range of pixel intensities; in order to calculate an adjusted exposuretime; and (d) capturing a second image of the subject at the adjustedexposure time thereby obtaining an image with an improved dynamic range.In a preferred process of the invention, the dynamic range isautomatically improved to an optimal dynamic range. In anotherembodiment of the invention, the process further comprises ensuring thatno more than a predetermined threshold number of the image pixelintensity values are saturated. In general, the predetermined thresholdnumber of image pixel intensity values, which is found to be saturated,falls in the range of about 0.0% to about 0.05%. Ideally no more thanabout 0.02% of the image pixel intensity values are saturated.

In another embodiment, step (c), above, further comprises determiningthe adjusted exposure time as a function of the center of mass and amidpoint of the histogram. Preferably, the adjusted exposure timeprovides an image in which the center of mass of the histogram coincideswith the midpoint of the histogram. Also, the process of the inventionincludes iteratively conducting the steps (a)-(d), above, to provide animage having an improved or optimal dynamic range. For example,conducting the steps iteratively ceases when the difference between theexposure time and the adjusted exposure time is less than apredetermined tolerance. Preferably, the number of iterations does notexceed 150 iterations, more preferably, the number of iterations doesnot exceed 30 iterations.

In one embodiment, the predetermined tolerance is determined by theequation: |E/E′−1|<T, where E is the exposure time, E′ is the adjustedexposure time and T is a predetermined tolerance. In one case, forexample, T may be less than 0.25. In another aspect of the invention,the disclosed process further comprises flagging or otherwiseidentifying an image if after iteratively improving the dynamic range,the center of mass does not coincide with the midpoint of the histogram.What is more, such flagging of the image can be used to remove it from adataset used for further analytical evaluation. A preferred process mayfurther comprise adjusting the exposure time to reduce the number ofimage pixel intensity values that are saturated by capturing a new imageat one half the first exposure time. More preferably, the exposure timeis adjusted iteratively until the center of mass corresponds to themidpoint of the histogram.

In another embodiment, a process is described in which the exposure timeis adjusted to reduce the number of image pixel intensity values thatare saturated by capturing a new image at a new exposure time that isproportionally lower than the first exposure time based on the number ofsaturated pixel intensity values. Once again, the exposure time can beadjusted iteratively until the center of mass corresponds to themidpoint of the histogram. The exposure time can also be adjusted toreduce the number of image pixel intensity values that are saturated bycapturing a new image at a reduced, or minimum exposure time in the stepof capturing an image.

While the subject of the invention may include any subject that isamenable to providing an image, particularly in the context ofpharmaceutical, biological and medical research, a preferred subjectincludes a biological specimen, a biological tissue specimen, a wholetissue section, a tissue microarray, or combinations thereof. Thesubject may further include a tissue section, preferably one that hasbeen stained, more preferably with a fluorescent substance. In oneembodiment, an image is captured by the camera through a microscope. Theprocess disclosed may also be applied to capturing a plurality of suchimages, such as tissue “histospots” contained in a tissue microarray.

Accordingly, a process of normalizing quantitative data across multipleslides, instruments and operators is described, which process includes:(a) capturing a first image from a first sample at a first exposuretime, resulting in a captured first image comprising a predeterminednumber of pixels each pixel having an intensity value; (b) querying afirst frequency distribution of pixel intensities values of the capturedfirst image; to determine a center of mass of the first frequencydistribution to determine a second, adjusted exposure time; (c)capturing a second image of the first sample at the second, adjustedexposure time thereby improving the dynamic range of the second imageover the first image; (d) capturing a third image from a second sampleat a third exposure time, resulting in a captured third image comprisinga predetermined number of pixels each pixel having an intensity value;(e) querying a second frequency distribution of pixel intensities valuesof the captured third image to determe a center of mass of the secondfrequency distribution to determine a fourth, adjusted exposure time toprovide a fourth image having a dynamic range corresponding to thedynamic range of the first captured image; and (h) capturing a fourthimage from the second sample at the fourth, adjusted exposure time.

A computer-usable medium is also provided having computer readableinstructions stored thereon for execution by a processor to perform aprocess for automatically adjusting the exposure time associated with acaptured image, wherein the instructions comprise steps performediteratively of: (a) capturing an image of a subject at a first exposuretime, resulting in a captured image comprising a predetermined number ofpixels each pixel having an intensity value; (b) querying a frequencydistribution of pixel intensity values; (c) determining a center of massof the frequency distribution of pixel intensity values to determine anadjusted exposure time; and (d) capturing a subsequent image of thesubject at the adjusted exposure time thereby obtaining an image with anoptimal dynamic range.

The invention also encompasses an electromagnetic signal carryingcomputer-readable instructions stored thereon for execution by aprocessor to perform a process for automatically adjusting an exposuretime to optimize a dynamic range of a digital image includes: (a)capturing an image of a subject at a first exposure time, resulting in acaptured image comprising a predetermined number of pixels each pixelhaving an intensity value; (b) querying a frequency distribution ofpixel intensity values of the pixels verses intensity values; (c)determining a center of mass of the frequency distribution of pixelintensity values to determine an adjusted exposure time; and (d)capturing a second image of the subject at the adjusted exposure timethereby obtaining an image with an optimal dynamic range.

Moreover, a system is provided for automatically adjusting an exposuretime to improve or optimize a dynamic range of a digital imagecomprising: a camera or image sensor configured to capture an image of asubject within the field of view at a first exposure time, resulting ina captured image comprising a plurality of pixels, each pixel having anintensity value; a shutter configured to control the exposure time ofthe camera; and a controller configured to carry out the stepscomprising: (a) querying a frequency distribution or histogram of pixelintensity values of the pixels versus intensity values; (b) determininga center of mass of the histogram to determine an adjusted exposuretime; and (c) capturing a second image of the subject at the adjustedexposure time thereby obtaining an image with an improved or optimaldynamic range.

Additional embodiments will become apparent to those of ordinary skillin the art from the following, additional detailed descriptions. Forinstance, although applied to whole tissue sections and TMA's in theexamples which follow, the present technique has other potentialapplications. When performing quantitative analysis of digital images,the presence of saturated pixels results in an overall underestimate ofintensity for both that individual pixel and its contribution to thecollection of pixels being studied. Conversely, the undersaturation ofan image can potentially concentrate a range of intensities within alimited number of histogram bins (as defined by the bit depth of thecamera being used), thus inhibiting the discrimination of individualpixels and impacting overall dynamic range. The processes describedherein provide two key features: to mitigate impact of saturated pixelsand to center the image pixel intensity distribution in the dynamicrange. Both of these features are critical in any situation wherequantitative information is desired from an acquired image. Thealgorithms are wavelength independent and can be adapted to suit a rangeof experimental and hardware conditions (i.e., saturation thresholds,camera bit depth).

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages of theinvention will be apparent from the following more particulardescription of preferred embodiments of the invention, as illustrated inthe accompanying drawings in which like reference characters refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, where emphasis is instead placed upon illustratingthe principles of the invention.

FIG. 1 illustrates a block diagram of an exemplary imaging microscopesystem.

FIG. 2 illustrates a flow diagram of an embodiment of a procedureembodying the AutoExposure technique.

FIG. 3A through FIG. 3B illustrates regression analysis of datagenerated from images acquired at single exposure time (y axis) andusing the AutoExposure techniques described herein (x-axis) for each oftwo common IHC stains.

FIG. 4A and FIG. 4B illustrates clinical outcome data generated fromimages of ER stained specimens obtained using single exposure time (A)and the AutoExposure techniques described herein (B).

FIG. 5A and FIG. 5B illustrates clinical outcome data generated fromimages of PR stained specimens obtained using single exposure time (A)and the AutoExposure techniques described herein (B).

FIG. 6A through FIG. 6D illustrates regression analysis assessingreproducibility of data (AQUA® score) generated from images acquiredusing the AutoExposure techniques described herein (A), and thereproducibility of AutoExposure techniques described herein dictatedexposure time to acquire images in the fluorescent channel for DAPI (B),Cy3 (C) and Cy5(D).

FIG. 7 illustrates tabulated results of an exemplary analysis using theAutoExposure techniques described herein.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

A description of the preferred embodiments of the invention follows.

Referring to FIG. 1, an exemplary reflected-light fluorescent microscopesystem 100 includes a an excitation source 102, an objective lens 104, asample supporting stage 106, a filter assembly 115 including one or morefilters 108′, 108″, 108′ (generally 108), and an observation head 110.The sample supporting stage 106 is configured to support a sample undertest along an optical axis and within a focal plane of the objectivelens 104. One of the filters 108 of the filter assembly 115 is alsolocated along the optical axis between the objective lens 104 and theobservation head 110. In the exemplary embodiment, each filter 108 is athree port device with two opposite ports disposed along the opticalaxis and a third port disposed off-axis. As illustrated, the third portcan be orthogonal to a line joining the two opposite ports.

Illumination from an excitation source 102 is directed toward theorthogonal port of the filter 108. The filter 108 redirects a portion ofthe illumination from the excitation source 102 toward the objectivelens 104. The objective lens 104 preferably includes a relatively highnumerical aperture thereby allowing it to capture a substantial portionof excitation light. The objective lens 104 functions as a condenserdirecting excitation light toward a sample under test placed upon thestage. In some embodiments, multiple objective lenses 104 (e.g., 4×,10×, 20×, 40×, 60×) are included within a single nosepiece (not shown).The nosepiece can be manipulated to selectively bring different ones ofthe multiple objective lenses 104 into alignment with the optical axisto adjust magnification of the sample under test.

Sample illumination (emission) from the sample under test travels backalong the optical path through the objective lens 104 and into a firstone of the opposite ports of the filter 108′. At least a portion of thesample illumination continues along the optical path, exiting a secondone of the opposite ports of the filter 108′ directed towards theobservation head 110. As described in more detail below, the filter 108′selectively filters the sample illumination passed therethrough. Influorescence microscopy, filtration can be used to selectively viewemissions from different fluorophores (e.g., red, green, blue) used totreat the sample. As illustrated, the microscope system 100 can includemultiple filters 108′, 108″, 108′″, each filter 108 tuned to pass aselected wavelength of the sample emission toward the observation head110. The different filter blocks 108 when housed within a carousel orturret 115, allow for rapid selection of each filter 108 withoutdisturbing the sample under test. In some embodiments, the differentfilters 108 are radially disposed within the turret 115 about an axis ofrotation. The turret 115 is positioned with its axis of rotationparallel and to a side of the optical axis, such that one of the filters108′ is aligned with the optical axis. Rotation of the turret 115selectively moves one filter block 108′ out of alignment and bringsanother one of the filter blocks 108″, 108′″ into alignment with theoptical axis.

The observation head 110 directs at least a portion of light from thefilter block 108 toward an image collection device, such as a chargecoupled device (CCD) camera 112. In some embodiments, the observationhead 110 additionally includes one or more eyepieces (not shown)allowing for manual observation of the sample under test. Such aneyepiece can be used to adjust placement of a sample 107 upon the stage106 and to coordinate positioning of the stage 106 before and duringtest. In some embodiments, a first shutter 117 is provided to controlexposure time of the sample 107 to the excitation source 102. A secondshutter 114 is provided to control exposure time of an imaging device,such as the CCD camera 112. As shown, the shutter 114 can be anindependent component located along the optical path between the sampleunder test and the observation head 110. Alternatively or in addition toan independent shutter 114, the shutter can be integrated into the CCDcamera 112.

The microscope system 100 also includes a controller 116 for controllingthe overall image acquisition process. Preferably, the controller 116 isin communication with one or more sub-elements of the microscope system110 to allow automated control of the system. In the exemplaryembodiment, the controller 116 is in communication with the excitationsource 102, the objective lens 104, the CCD camera 112, the shutter 114,the turret 115, and a stage positional controller 118. The controller116 can include at least one microprocessor or computer 116 operatingunder the control of pre-programmed instruction or code.

The pre-programmed instructions may be stored on a computer-readablemedium, such as a magnetic disk (e.g., a hard disk drive), an electronicmemory (e.g., a RAM), or an optical media, such as an optical disk(e.g., a CD ROM). The pre-programmed instructions can be configured toimplement one or more of the procedural steps referred to herein.

In operation, the controller 116 sends a signal to the stage positionalcontroller 118 to position the stage 106, such that a selected for orhistospot 109 of the sample under test is brought into alignment withthe optical axis. The controller 116 also sends a signal to an axialtranslator 119 configured to position and reposition the objective lens104 along the optical axis with respect to the stage 106 bringing amagnified image of the sample into focus. For embodiments including amotorized nosepiece, the controller 116 sends a second signal to thenosepiece causing it to rotate a selected one of multiple objectivelenses 104 into alignment with the optical axis prior to focusing. Thecontroller 116 also sends a signal to the turret 115 causing acontrolled rotation of the turret to select one of the multiple filters118. In response, the turret 115 rotates, bringing the selected one ofthe filters 118 into alignment with the optical axis. The controller 116next sends a signal to the excitation source 102 turning the source 102on, at least momentarily, to illuminate the sample under test. The imagesensor, or camera shutter 114 is normally closed blocking the opticalpath between the sample under test and the CCD camera 112. For somemicroscopes the light source 102 is turned on during initialization ofthe instrument. With fluorescent microscopes, the high-intensity lampsrequire a warm-up period to allow intensity of the source 102 tostabilize before any test samples are measured.

For such fluorescent systems, the light source 102 remains on duringoperation. The illuminated source or lamp shutter 117 between lightsource 102 and test sample is used to block illumination of the sampleuntil ready to view the sample and acquire an image of the sample. Suchlimited exposure of the test sample to illumination may avoid bleachingof the sample. Upon receiving a first trigger signal from the controller116, the shutter 117 opens for a predetermined exposure period beforeclosing again. A second trigger signal from the controller is sent tothe camera shutter 114. This signal controls exposure thereby allowingfor a controlled sampling by the image sensor 112 of emission from thesample under test 107. In some embodiments, the first shutter 117 isopen for at least the entire duration of an exposure controlled by thesecond shutter 114. In some embodiments, operation of the two shutters114, 117 can be controlled by a common signal, or otherwise configuredto operate in synchronization. Under control of the controller 116, theCCD camera 112 captures an electronic image of illumination from thesample under test. The image can be forwarded to the controller 116 orto an external system for analysis.

With optional independent control of the two shutters 114, 117, timingof each shutter can be varied to produce different effects. For example,in some embodiments, the first shutter 117 is opened to expose testsample for a predetermined period of time and then closed. This can beperformed to expose a luminescent test sample to illumination from thesource 102. The second shutter 114 could be operated after closure ofthe first shutter 117 to sample luminescence of the sample, withoutinterference from source illumination.

During typical operation of the system, a microscope mounted tissuesample in either tissue micro array (TMA) or whole tissue section (WTS)format treated with immunoassay reagents and detection fluorescenthistochemical stains to delineate morphological or quantitative featuresof the sample (i.e., regions of tumor, sub-cellular compartments and thetarget biomarker) is analyzed. The stained slide 107 is loaded onto theautomated microscope platform 106 and multiple low magnifications images(4×) of a limited field of view are collected to generate a mosaic imageof the active region of interest for analysis and determine acquisitionparameters for the high resolution acquisitions. Among basic systemsetup parameters, the most important setup condition is the exposuretime for the acquisition of an image in each channel for each filed ofview required for the experiment.

In use, a test sample may include a TMA having a matrix of a tissue orcell line samples on a single microscope slide. The test sample may alsobe a whole tissue section, for example, from a formalin fixed, paraffinembedded sample block. Also, combinations of TMAs and whole tissuesections may be measured on the same slide.

In examples described herein, a TMA comprised of breast cancer tissuespecimens with a range of expression (normal, low, medium and high) wasused as a test sample for the automated exposure processes. Individualsections taken from TMAs (designated 40_7, Yale TMA facility, New Haven,Conn.) on microscope slides were put through a typical stainingprocedure as prescribed for quantitative AQUA® analysis.

The staining procedure comprises the following steps. Sections weredeparafinized in xylene, rehydrated through a series of decreasingamounts of ethanol to pure water, and subjected to antigen retrieval inTris EDTA. After endogenous peroxidase blocking and blocking withBackground Sniper (Biocare Medical, Concord, Calif.), primary antibodiesto: HER 2, polyclonal (Dako, Carpinteria, Calif.), Estrogen receptor(ER), Clone 6F11 (Novocastra, Laboratories Ltd, Burlingame, Calif.) orProgesterone Receptor (PR) Clone PgR636 (Dako) and compartment specificantibody to cytokeratin (Rabbit, Dako) each diluted in Da Vinci Green(Biocare Medical) were applied and rinsed off after one hour. Envisionanti-mouse (Dako) and Alexa 555 GAR (Molecular Probes, Eugene Oreg.)were then applied. After extensive washing, TSA Cy 5 tyramide (NEN,Perkin Elmer, Waltham, Mass.) was applied. The slides were then washedin TBS/Tween 20. Finally, Prolong Gold antifade reagent with DAPI(Molecular Probes) mounting media was applied and the slides were dried.

After slides were stained, digital images for each TMA histospot wereacquired on an AQUA® analysis PM-2000™ system, commercially availablefrom HistoRx, Inc. of New Haven, Conn. Images in each relevantfluorescence channel were collected using filters for Cy5 (for ER, PR),Cy3 (for cytokeratin) and DAPI).

Software to perform image acquisition, entitled AQUASITION (version 2.0build 2) was loaded on the PM-2000™ system. The software incorporatednecessary tools for image acquisition with the feature toactivate/deactivate automated exposure time calculations.

In one particular embodiment, a fluorescent microscope system is part ofan integrated quantitative immunohistochemical (IHC) analysis system,such as the PM-2000™ system. The IHC analysis system comprises thefollowing components assembled in a light-tight enclosure: a fluorescentmicroscope, such as the Olympus BX51 epi-fluorescence microscope,commercially available from Olympus America, Inc. of Center Valley, PA;the microscope is equipped with a motorized nosepiece to controlselection among different objective lenses (e.g., 4×, 10×, 20×, 40×,60×), and a motorized filter turret to control selection among differentfilter cube selection (e.g., in DAPI, Cy2, Cy3, Cy5 and Cy7 orequivalent wavelengths). The system also includes a motorized stage,such as the Prior Scientific part no. H101A. The PCI card that drivesthe stage is Prior Scientific part no. H252 motorized stage commerciallyavailable from Prior Scientific, Inc. of Rockland, Mass. The controlcard occupies a PCE expansion slot within a computer controller. Focuscontrol is facilitated by integrated software. The system also includesa light source, such as the X-CITE 120 system, commercially availablefrom EXFO Life Sciences & Industrial Division of Ontario, Canada, whichis equipped with a mercury/metal halide lamp; a monochromatic digitalcamera for images capture, such as the QUANTIFIRE camera, commerciallyavailable from OPTRONICS of Goleta, Calif.; and a computer controller.In the exemplary embodiment, the computer is a personal computer runningWINDOWS XP or higher operating system environment.

Each experimental slide or specimen was acquired first using traditionalsingle exposure methods. Each slide was subsequently acquired usingAutoExposure techniques described herein. According to a preferredembodiment, the acts of acquiring an image at optimized AutoExposuresettings is performed as described in the flow chart shown in FIG. 2. Animage is captured at a current exposure time in step 210. The currentexposure time can be chosen as a pre-selected for each newly capturedimage (210). Such a pre-selected value can be a constant for everyimage. In some embodiments, the constant, pre-selected exposure time isselected to promote overexposure for the first captured image.

A distribution of the intensity values for each pixel of the capturedimage is queried in step 220. In some embodiments, the pixel intensitydistribution is a frequency distribution. The frequency distribution canbe generated for example as a histogram, in which bins of the histogram(the x-axis) correspond to different intensity values, and the y-axiscorresponds to the number, or frequency, of pixels of the captured imagecounted in each of the different bins. For an exemplary 8-bit pixel, thehistogram could be configured to have 256 bins, with each bincorresponding to a distinguishable different pixel intensity rangingfrom a minim value (i.e., 0) to a maximum value (i.e., 255).

A measure of the amount of pixel saturation, for example as a ratio ofthe number of saturated pixels to the number of total pixels, iscompared to a threshold saturation value in step 230. In someembodiments, the threshold saturation value is selectable, as may beadvantageous given the particular application. Saturated pixels can beidentified as pixels having a maximum allowable value (e.g., anintensity value of 255 for an 8 bit pixel intensity scale of 0-255). Ifthere are too many saturated pixels, the image may be overexposed.Beneficially, such over exposure can be corrected by reducing theexposure time and re-capturing the image using the reduced exposuretime.

In some embodiments, a reduced exposure time is selected in response todetermining that the captured image is overexposed. Such selection ofthe reduced exposure time can be determined as a function of the amountof over exposure. Alternatively or in addition, such selection of thereduced exposure time can be determined according to a selectableaggression value—more aggressive resulting in coarser steps of reducedexposure time, less aggressive resulting in finer steps. In someembodiments, the exposure time may be reduced by up to one-half of itscurrent value based on the amount of overexposure in step 240 (thiswould set an upper limit to the coarseness of steps).

An image of the same sample is recaptured using the reduced exposuretime at step 241. Steps 220-230 are then repeated using the new imageobtained with the reduced exposure time. If the number of saturatedpixels determined in step 230 is less than the predefined thresholdsaturation value, the image is determined as not being overexposed. Ifthe image is not overexposed and maximization of the dynamic-range,otherwise referred to as the COM analysis herein, has not been executed,then an effective “center of mass” (COM) of the pixel distribution (orhistogram when available) is found in step 250. If the COM issufficiently close to the mid-point or center of the full range of pixelintensities (i.e., the dynamic range) then the image is saved in step260. If the COM is not sufficiently close to the mid-point or center ofthe dynamic range, a corrected exposure time is determined in step 270.In at least some embodiments, the corrected exposure time is determinedas a function of the distance of the COM from the mid-point or center ofthe possible pixel intensity frequency distribution. Then, steps 210-230are repeated. At that point, if the image is not overexposed and the COManalysis has been executed, the image is saved in step 260. If the imageis overexposed, then steps 240, 210-230 can be repeated.

AutoExposure may be a process by which the exposure time of the cameraimaging tissue micro array (TMA) histospots and whole tissue section(WTS) fields of view (FOV) are adjusted automatically via a softwarealgorithm. One aspect of the algorithm is to set the exposure time ashigh as possible without overexposing the image. This characteristicwill produce images that have a maximized dynamic range with minimalloss of information caused by exposure time.

As shown in FIG. 2, the overall process can include multiplesub-procedures:

A first sub-procedure relates to an overexposure correction procedure.If the current exposure time results in an overexposed image, a reduceexposure time is determined to reduce and in some instances eliminatethe amount of overexposure. This sub-procedure can be repeatediteratively until the image is no longer overexposed, or at least untilany overexposure is below an acceptable threshold. This overexposurecorrection procedure yields a sample image exhibiting minimalinformation loss.

A second sub-procedure relates to improving a dynamic range for thetones or pixel intensities of the sample image. This sub-procedureeffectively spreads or otherwise adjusts the frequency distribution ofpixel intensities of the image over a wider dynamic range by moving aso-called “center of mass” (COM) of the frequency distribution towards acenter of the possible range of pixel intensities thereby maximizing thedynamic range for the sample image.

Of course the “center of mass” as used herein does not literally referto usual meaning relating to a weighted average of mass with respect todistance. Rather, the concept of a COM is used in an analogous mannerreferring to the pixel intensity or tonal distribution. Considering apixel intensity histogram in which each bin is a measure of pixelintensity (tone), the bin value corresponds to a distance value withinthe standard meaning of COM. The bin count, or frequency, identifyingthe number of pixels having a particular intensity, or range ofintensities, corresponds to a mass value within the standard meaning ofCOM. Thus, the COM of a pixel intensity distribution provides a weightedaverage of the number of pixels having a given intensity value (i.e.,bin) versus the value of the bin. This analogy is useful in identifyingthat portion of the pixel intensity distribution in which a largernumber of pixels of any given image can be found. It is this regionhaving the proportionally larger number of pixels that can be movedtoward a mid-point of the possible range of pixel intensities to improvethe dynamic range of a given sample image.

In some embodiments a third sub-procedure, the overexposure correctionprocedure (i.e., the first sub-procedure) can be run again to ensurethat the dynamic-range corrected image is not overexposed.

An image can be considered overexposed if a percentage of the pixels inthe image have an intensity equal to the maximum resolvable intensityexceeds a predetermined threshold number, also known as the saturationlimit. This predetermined threshold number may be about 0.05%,preferably, 0.02%, more preferably 0.0%.

There are various ways to correct for overexposure. One is to acquire anew image at half the exposure time of the previous image. If necessarythis can be done reiteratively until saturated pixels are minimized.This allows for a quick adjustment in exposure time to bring the pixelintensities down within the range of detection to optimize exposure anddynamic range. However this simplistic approach may also cause thesystem to overcorrect for saturated pixels and set the new exposure timetoo low. Therefore it is desirable to modify the aggressiveness of thecorrection to the exposure time to be proportional to how many pixelsare saturated in the previous image. To achieve this the new exposuretime may be calculated as:

$\begin{matrix}{{E = {E^{\prime}{x\left( {1 - (0.5)^{({1 + S})}} \right)}}},} & (I) \\{where} & \; \\{S = {A\frac{{CCD}_{x}{CCD}_{y}{SL}}{P}}} & ({II})\end{matrix}$

where E is the new exposure time, E′ is the currently set exposure time,A is an aggression level, SL is the saturation limit, CCD_(x) andCCD_(y) represent the pixel dimensions of the captured image, and P isthe count of pixels at maximum intensity. The aggression level, A, mayvary but, generally, the values that one would want to choose woulddepend upon the amount by which images tend to be over saturated. Avalue of zero (0) for A represents a minimum value for which theexposure time would be halved. A practical maximum value for A is about10, after which the exposure time will not change enough for thealgorithm to be useful. In a preferred embodiment of the invention, thevalue for A can fall in the range of about 0<A<4.5. More preferably, Ais set at about 3.5.

The procedure of reducing exposure time to ensure the image is notoverexposed is a multi-step process. In an exemplary embodiment, a 256bin histogram is generated first for an 8-bit per pixel image obtainedfrom the camera at the current exposure time, E′. The number ofsaturated pixels are identified and compared to a predeterminedsaturation threshold value. Then, if the image is at or below thesaturation limit, the over-exposure procedure is exited. However, if theimage is over exposed, the exposure time is decreased. The new,decreased exposure time can vary based upon the number of currently overexposed pixels. In an exemplary embodiment, a value S can be determinedas

$\begin{matrix}{{S = {A\frac{0.0002 \times 2048^{2}}{M}}},} & (1)\end{matrix}$

in which A is an “aggression level” currently defined at 3.5 and M isthe count of pixels at maximum intensity. Then, the next exposure time Eis derived as follows:

E=E′−E′0.5^(1+S).

When the number of over exposed pixels is much greater than thesaturation limit, E≈E′−0.5E′ (i.e., the exposure time would be halved).The minimum amount of change to the current exposure time occurs whenthe number of over saturated pixels is very nearly equal to thesaturation limit, in which case E≈E′−0.088E′. Thus, because thealgorithm is exited when the image is at or below the saturation limit,the number of saturated pixels will never equal the saturation limit.The procedure of reducing exposure time can be repeated in an iterativemanner until the amount of overexposure is within a chosen threshold, oruntil a maximum number of iterations has been accomplished. In eitherinstance the over-exposure correction routine is exited.

An alternative and equally viable process for correcting foroverexposure is to acquire a new image at a minimum exposure time, thenproceed with optimizing the exposure time by calculating the COM andbringing it within range of the midpoint, as described above.

There are circumstances in which it may not be possible to correct foroverexposure. For example in fluorescent microscopy, various debris maybe unintentionally caught in the mounted tissue sample causing unwanted,intense fluorescence, even at minimal exposure times. Application of theprocesses described herein are beneficial in recognizing such sampleshaving this problem. Such samples can be flagged or otherwise identifiedfor subsequent review, or identified as erroneous, and thus excludedfrom subsequent image data analysis. To achieve this the number ofreiterations the system carries out can be limited. After a limitednumber of attempts to correct one or more of the exposure and thedynamic range, the sample can be flagged. Generally, the number ofiterations x can fall in the range of about 5≦x≦150. In a preferredembodiment of the invention, the number of iterations x is set to fallin the range of about 10≦x≦30.

Dynamic Range Optimization

The following process can be implemented to optimize the dynamic rangeusing a “center of mass” technique.

1. Capture an image of the specimen, the image made up of many pixels,each pixel having a signal intensity value.

2. Query a pixel intensity distribution of the captured image. In atleast some embodiments, this step includes generation of a histogram ofthe image for intensity (with a full scale i.e., entire dynamic rangeavailable) versus number of pixels.

3. Calculate the center of mass (COM) of the histogram data. Moregenerally, this step involves identification of that portion of thequeried pixel intensity distribution including a relatively large numberof pixels.

4. Determine an adjusted exposure time that will achieve an image with acenter of mass at the midpoint of the histogram (and midpoint of dynamicrange).

5. Re-acquire an image of the specimen at the adjusted exposure time.

By way of example, a 256 bin histogram is generated of an 8 bit perpixel image from the camera at the current exposure time, E′. If theimage is overexposed, the algorithm is exited. The current center ofmass, C, of the histogram can be determined as follows:

$\begin{matrix}{{C = \frac{\sum\limits_{i = 1}^{N}\; {iH}_{i}}{\sum\limits_{i = 1}^{N}\; H_{i}}},} & (3)\end{matrix}$

in which N is the number of histogram bins (256) and H_(i) is the countof items in the i^(th) bin. The new exposure time can be calculated byattempting to shift the current C to the center of the histogram. Thevalue of C can be compared to a center or midpoint of the pixelintensity distribution (histogram). If the current C falls after themidpoint (N/2) of the histogram, the adjusted exposure time can bedetermined as follows:

$\begin{matrix}{{E = {\sqrt{\frac{\frac{1}{2}N}{C}}E^{\prime}}},} & (4)\end{matrix}$

otherwise, the adjusted exposure time can be determined as follows:

$\begin{matrix}{E = {\sqrt{\frac{C}{\frac{1}{2}N}}{E^{\prime}.}}} & (5)\end{matrix}$

If |E/E′−1|<T, where T is a tolerance currently set at a thresholdvalue, such as 0.25, then exit the algorithm. The above steps in the“center of mass” technique are repeated until the difference between theexposure times is within the chosen tolerance (e.g., 0.25).

Since the above techniques can be automated, being performed by aprocessor, such as the controller 116 (FIG. 1), executing preprogrammedinstructions to process a pixelized image of a sample, they can beexecuted much more quickly and reliably than if done manually by anoperator. In at least some embodiments, the time savings is sosubstantial, that the AutoExposure process can be repeated for every FOV(histospot), while still completing analysis of a specimen within areasonable time period. The ability to perform AutoExposure on a FOV toFOV (histospot to histospot) basis is dependent upon the expression ofall pixel values in an image in terms of a measure of signal power,instead of signal value. By performing all image analyses with respectto this signal power, images with different exposure times can benormalized and data obtained from such images can be directly compared.The power P is obtained by diving the raw pixel value from the camera bythe exposure time and the current bit-depth (i.e., 256 and 4096). Thisis an expression of the percent of maximum measurable intensity hittinga given pixel in the camera every millisecond and is given by

$\begin{matrix}{{P = \frac{I}{256t}},} & (6)\end{matrix}$

where I is the measured pixel intensity, t is the exposure time (inmilliseconds), and 256 is the bit-depth. Any bit-depth, such as 4096,can be used with no loss of generality. The resulting value has units ofms⁻¹. Note that this is not a classical definition of power (energy pertime) because the wavelength and frequency of light captured by thecamera is not considered.

In some embodiments, the overexposure correction process (e.g., steps210 through 241 of FIG. 2) can be performed for an image withoutperforming a COM analysis. Similarly, in some embodiments, the COMprocess (e.g., step 210 and steps 250-271) can be performed for an imagewithout performing an overexposure correction process. When bothoverexposure and COM analyses are performed for a given image, they maybe performed in either order, with one or more of the individualprocedures being repeated in any sequence as may be beneficial inanalyzing a specimen. Subsequent to application of the correctivetechniques described herein, the adjusted image can be further analyzedusing any available image analysis techniques. In at least someembodiments, further analysis includes a quantitative analysis of thesample image, such as provided by the AQUA® analysis, yielding anAQUASCORE for a given sample image.

Post Acquisition Validation

For both single exposure and AutoExposure acquired slides, each tissuespot was qualitatively assessed for focus and gross staining artifacts,followed by automated validation for split-spot images, percent tumorareas (spots with less than the 5% tumor as assessed by total pixelcount were redacted), % saturation (spots with greater that 5%saturation for DAPI, 4% saturation for Cy3, and 1% saturation for Cy5were redacted), sum-channel (total) intensities (bottom 10% for bothDAPI and Cy3 were redacted) and finally, the ratio of DAPI nuclearsignal to DAPI signal in cytoplasm (spots less than 1.5 were redacted).It is important to note that as a critical endpoint assessment forAuto-Exposure, no tissue spot percentage saturation exceeded the maximumof 0.02% for either array.

Data Analysis

For each validated tissue cohort, several analyses of the acquiredimages were done in order to assess the functionality, reproducibility,and utility of AutoExposure. Regression, distribution, and survivalanalysis (both Kaplan-Meier and Cox univariate) were performed usingSPSS v.15.01 (SPSS, Inc; Chicago, Ill.). For survival analysis, datacutpoints were generated by a two-step cluster algorithm (SPSS) based onlog-likelihood distance using Bayesian criterion as correction.

RESULTS Correlation Of Auto-Exposure With Single Exposure Time Setting

To assess how AutoExposure functions with respect to AQUA® scoregeneration, linear and non-parametric regression analysis (FIG. 3A-FIG.3B) was performed comparing AQUA® scores between single exposure timeacquisition by an experienced operator and AutoExposure acquisition.AQUA® scores for TMA histospots stained for ER (FIG. 3A), and PR (FIG.3B) all showed a highly significant correlation, both Pearson R andSpearman's Rho, greater than 0.9. These results indicate that, atminimum, AutoExposure functions equivalently to a trained, experienceduser setting a single exposure time.

Auto-Exposure Functionality With Respect To Clinical Outcome

To confirm the above linear regression analysis and to ascertain whetherAutoExposure has an affect on the data and therefore the resultingoutcome prediction, an assessment was made on the overall patientsurvival for each of the markers tested based on data obtained usingsingle exposure time acquisition and AutoExposure. Data cutpoints weregenerated by two-step clustering for each marker and exposure procedureas described above.

Analysis of the data for ER (FIG. 4A-FIG. 4B) showed that clusterassignments were virtually identical and survival outcome also showedthe same non-significant trend for data obtained using single exposuretime (FIG. 4A) and AutoExposure (FIG. 4B). Analysis of the data for PR(FIG. 5A-FIG. 5B), again showed similar cluster assignments and theresulting survival analysis revealed a significant (p=0.023 and 0.005for single and AutoExposure respectively) association between highexpression and decreased five-year disease specific survival for bothsingle exposure time (FIG. 5A) and AutoExposure (FIG. 5B). Takentogether, these data strongly support that AutoExposure is equivalent toan experienced operator setting a single exposure time manually.

Distribution Analysis

An important consideration in all immunoassays is dynamic range. Todetermine the affect of AutoExposure on dynamic range, a frequencydistribution analysis was performed examining the mean, standarddeviation, variance and range.

For data obtained from ER and PR stained samples, an increase in each ofthese metrics was observed as shown in Table I of FIG. 7 for 1) allsamples combined, 2) samples with AQUA® scores in the lowest 50%, and 3)samples with AQUA® scores in the highest 50% with AutoExposure comparedto the single-exposure time indicating AutoExposure increases thedynamic range of quantitative IHC data over that obtained using singleexposure. The results demonstrated an increase in dynamic range for ERand PR quantitative assays. These data support that AutoExposure addsfunctionality to existing technology by expanding dynamic range. Thisallows for increased resolution of quantitative IHC measurements such asAQUA® scores.

Reproducibility Of AutoExposure

In order to test the reproducibility of AutoExposure, images of the TMA(YTMA49) stained with ER were re-acquired with AutoExposure, aspreviously described. FIG. 6A-FIG. 6D shows linear and non-parametricregression analysis of AQUA® scores (FIG. 6A), DAPI exposure times (FIG.6B), Cy3 exposure time (FIG. 6C), and Cy5 exposure times (FIG. 6D). Allshowed significant and robust correlation. These data demonstrate, thatin addition to being highly correlative with single exposure timesetting, AutoExposure is highly and significantly reproducible untoitself.

As a further benefit, it was initially thought that in order tostandardize the results of AQUA® analysis, machine to machine variationwould need to be addressed, along with operator to operator variance andslide to slide variability, not to mention day-to-day variations inrunning the same assays. It was thought that assay variability could beattributed to many factors, including the inherent variations introducedby enzymatic assays, which utilize amplification steps that are affectedby even slight differences in incubation time. Hence, thereproducibility of results could vary from slide to slide. It has beendiscovered, however, that quite surprisingly the techniques describedherein do away with the need for common controls to normalize resultsacross slides. The AutoExposure techniques described herein seem toaccomplish standardization all on its own, although how it does so isnot yet well understood.

While this invention has been particularly shown and described withreferences to preferred embodiments thereof, it should be apparent thatunique operational features have been described. Although particularembodiments have been disclosed herein in detail, this has been done byway of example for purposes of illustration only, and is not intended tobe limiting with respect to the scope of the appended claims whichfollow. In particular, it is contemplated by the inventors that varioussubstitutions, alterations, and modifications may be made to theinvention without departing from the spirit and scope of the inventionencompassed by the appended claims

1. A method for automatically improving a dynamic range of pixelintensities in a digital image comprising: (a) capturing with a firstexposure time a first image of a subject; (b) determining a region of afrequency distribution having a greatest frequency of occurrence ofpixel intensities; and (c) determining an adjusted exposure timeconfigured to move the region of the frequency distribution having thegreatest frequency of occurrence toward a center of the frequencydistribution when used to acquire an image of the same subject capturedwith the adjusted exposure time thereby obtaining an image having animproved dynamic range of pixel intensities.
 2. The method of claim 1,wherein the dynamic range is automatically improved to an optimaldynamic range.
 3. The method of claim 1, further comprising identifyinga number of saturated pixel intensities and reducing the exposure timeresponsive to the number of saturated pixels being above a predeterminedthreshold number.
 4. The method of claim 3, further comprising adjustingthe exposure time to reduce the number of image pixel intensity valuesthat are saturated by capturing a new image at one half the firstexposure time.
 5. The method of claim 4, wherein adjusting the exposuretime is done iteratively until the region of the frequency distributionhaving the greatest frequency of occurrence corresponds to the center ofthe frequency distribution.
 6. The method of claim 3, further comprisingadjusting the exposure time to reduce the number of image pixelintensity values that are saturated by capturing a new image at a newexposure time that is proportionally lower than the first exposure timebased on the number of saturated pixels.
 7. The method of claim 6,wherein adjusting the exposure time is done iteratively until the regionof the frequency distribution having the greatest frequency ofoccurrence corresponds to the center of the frequency distribution. 8.The method of claim 3, further comprising adjusting the exposure time toreduce the number of image pixel intensity values that are saturated bycapturing a new image at a minimum exposure time in the step ofcapturing an image.
 9. The method of claim 3, wherein the predeterminedthreshold number falls in the range of about 0.0% to about 0.05% of thetotal number of pixels in the plurality of pixels.
 10. The method ofclaim 9, wherein the predetermined threshold number is about 0.02% thetotal number of pixels in the plurality of pixels.
 11. The method ofclaim 1, wherein determining an adjusted exposure time further comprisesdetermining the adjusted exposure time as a function of a center of massand a midpoint of the frequency distribution of pixel intensities. 12.The method of claim 11, wherein the adjusted exposure time provides animage in which the center of mass of the frequency distribution of pixelintensities coincides with the midpoint of the frequency distribution ofpixel intensities.
 13. The method of claim 1, further comprisingiteratively conducting the steps (a)-(c), wherein the adjusted exposuretime is determined to provide an image having an improved or optimaldynamic range with each iteration.
 14. The method of claim 13, whereiniteratively conducting the steps ceases when the difference between theexposure time and the adjusted exposure time is less than apredetermined tolerance.
 15. The method of claim 14, wherein the numberof iterations does not exceed 150 iterations.
 16. The method of claim14, wherein the number of iterations does not exceed 30 iterations. 17.The method of claim 14, wherein the predetermined tolerance isdetermined by the equation: |E/E′−1|<T, where E is the exposure time, E′is the adjusted exposure time and T is a predetermined tolerance. 18.The method of claim 17, wherein the adjusted exposure time T is lessthan about 0.25.
 19. The method of claim 14, further comprisingidentifying the image if after iteratively improving the dynamic range,the center of mass does not coincide with the midpoint of the frequencydistribution of pixel intensities, whereby the identified image isremoved from a dataset used for further analytical evaluation.
 20. Themethod of claim 1, wherein the image is captured by an image sensorthrough a microscope.
 21. The method of claim 1, further comprisingcapturing a plurality of images, wherein each captured image has animproved dynamic range.
 22. The method of claim 21, wherein theplurality of images includes tissue histospots contained in a tissuemicroarray.
 23. The method of claim 1, wherein the pixel intensitiesrepresent signal power.
 24. A non-transitory computer-readable mediumhaving computer readable instructions stored thereon for execution by aprocessor, the instructions comprising steps performed iteratively of:(a) capturing an image of a subject at a first exposure time; (b)determining a center of mass of a frequency distribution of pixelintensity values to determine an adjusted exposure time; and (c)capturing a subsequent image of the subject at the adjusted exposuretime thereby obtaining an image with an optimal dynamic range.
 25. Asystem for automatically adjusting an exposure time to optimize adynamic range of a digital image comprising: a camera configured tocapture an image of a subject at a first exposure time; a shutterconfigured to control the exposure time of the camera; and a controllerconfigured to carry out the steps comprising: (a) determining a regionof a frequency distribution having a greatest frequency of occurrence ofpixel intensities; and (b) determining an adjusted exposure timeconfigured to move the region of the frequency distribution having thegreatest frequency of occurrence toward a center of the frequencydistribution when used to acquire an image of the same subject capturedwith the adjusted exposure time thereby obtaining an image having animproved dynamic range of pixel intensities.